Information provisioning apparatus and information provisioning method

ABSTRACT

A decoding method and apparatus are provided for decoding an audiovisual stream that is divided into a plurality of segments. The decoding method and apparatus obtain metadata that specify a segment out of the plurality of segments. The metadata is described in a structured description. The segment that is specified by the metadata is obtained, and a start time for rendering the segment from the metadata is derived. The segment is decoded based on the metadata to generate decoded segment data before the start time.

CROSS-REFERENCE PARAGRAPH

This is a continuing application of pending U.S. patent application Ser.No. 13/609,848, filed on Sep. 11, 2012, which is a continuation of U.S.patent application Ser. No. 13/165,111, filed on Jun. 21, 2011, now U.S.Pat. No. 8,555,328, issued Oct. 8, 2013, which is a continuation of U.S.patent application Ser. No. 12/899,860, filed on Oct. 7, 2010, now U.S.Pat. No. 7,992,182, issued on Aug. 2, 2011, which is a continuation ofU.S. patent application Ser. No. 12/111,021, filed on Apr. 28, 2008, nowU.S. Pat. No. 7,836,479, issued on Nov. 16, 2010, which is acontinuation of U.S. patent application Ser. No. 10/019,319, filed onJan. 10, 2002, now U.S. Pat. No. 7,383,566, issued on Jun. 3, 2008,which is a U.S. National Stage of International Application No.PCT/JP00/04736, filed on Jul. 14, 2000, which claims the benefit ofJapanese Application No. 11-200095, filed Jul. 14, 1999, the contents ofall of which are expressly incorporated by reference herein in theirentireties. The International Application was not published under PCTArticle 21(2) in English.

TECHNICAL FIELD

The present invention relates to an information provision apparatus,information receiving apparatus, and storage medium, and relates inparticular to an information provision apparatus, information receivingapparatus, and storage medium for video/audio, data, etc., operating viabroadcast media such as digital broadcasting and communication mediasuch as the Internet.

BACKGROUND ART

In recent years, there has been an active trend of digitalization ofbroadcasting, and fusion with communications has also progressed. In thefield of communications, satellite digital broadcasting has already beenstarted, and it is expected that terrestrial broadcasting will also bedigitalized in the future.

As a result of digitalization of broadcast content, data broadcasting isalso performed in addition to conventional video and audio. Also, in thecommunications field, digital content distribution via the Internet hasbegun with music, and Internet broadcasting stations that broadcastvideo have also appeared.

Henceforth, it is envisaged chat continuous content media such as videoand audio will enter the home via various paths (transmission media).Through such fusion and digitalization of communications andbroadcasting, it has become possible to offer previously unavailableservices by means of metadata that describes content or relates tocontent.

For example, EPG information as well as audio/video information isprovided by interleaving EPG (Electric Program Guide)—“Standardspecification for program arrangement information used in digitalbroadcasting ARIB STD-B10 Version 1.1” or “pr ETS 300 468 DigitalBroadcasting systems (or television, sound and dataservices—Specification for Service Information (SI) in Digital VideoBroadcasting (DVB) systems”) used in CS digital broadcasting, in anaudio/video PES (Packetized Elementary Stream) using an MPEG-2 (MotionPicturecoding Experts Group phase 2—“ISO/IEC 13818-1 to 3”) privatesection.

Also, in BS digital broadcasting, data broadcasting using MPEG-2 privatePES packets is anticipated. Moreover, it is also possible to performcontent management by inserting metadata that describes content in theformat of user data in material transmission (“ANSI/SMPTE291M-1996Ancillary Data Packet and space Formatting”).

A conventional information processing system will be described belowusing FIG. 15. FIG. 15 is a block diagram of a conventional informationprocessing system.

An information provision node 1501 is provided with a storage section1502 in which an AV stream and metadata for describing the AV stream arestored. Also provided in the information provision node 1501 is aninformation provision section 1504 that multiplexes the AV stream andmetadata stored in the storage section 1502 and generates and outputs amultiplex stream 1503. The information provision section 1504 transmitsthe multiplex stream 1503 to an information usage node 1506 via anetwork 1505.

Meanwhile, the information usage node 1506 is provided with aninformation usage section 1507 that extracts an AV stream and metadatafrom a multiplex stream and executes processing on them in order to usethem. The information usage node 1506 is also provided with a storagesection 1508 that stores the AV stream and metadata extracted by theinformation usage section 1507. The information usage section 1507 readsthe AV stream and metadata stored in the storage section 1508 in orderto use them.

Next, the information provision section 1504 will be described usingFIG. 16. FIG. 16 is a block diagram of a conventional informationprovision section.

The information provision section 1504 is provided with an accesssection 1601 that reads an AV stream and metadata from the storagesection 1502. The access section 1601 outputs an AV stream 1602 andmetadata 1603 to a multiplying section 1604.

The multiplexing section 1604 transmits to the information usage node1506 a multiplex stream 1503 that multiplexes the AV stream 1602 andmetadata 1603.

Next, multiplex stream generation processing by the multiplexing section1604 will be described using FIG. 17.

The drawing indicated by reference numeral 1503 in the drawing shows theMPEG-2 TS (Transport Stream) PES packet layer, and shows a multiplexstream. The drawing indicated by reference numeral 1702 shows a videoPES packet, the drawing indicated by reference numeral 1703 shows anaudio PES packet, and the drawing indicated by reference numeral 1703shows a private PES packet. 1603 indicates the metadata PES packetlayer, in which 1704 is a first PES packet comprising metadata and 1705is a second PES packet comprising metadata.

The multiplexing section 1604 divides the metadata 1603 to make privatePBS packets, inserts the first PES packet 1704 and second PES packet1705 in order as appropriate between AV streams consisting of video PESpackets 1701 and audio PES packets 1702, and obtains a multiplex stream1503 that is an MPEG-2 TS.

As conventional metadata is AV stream ancillary data—for example, smallamounts of data such as titles—processing has been performed withmetadata alone. That is to say, it has not been necessary to providetime synchronization of metadata with an AV stream. Therefore, sinceconventional metadata does not have a configuration that provides forsynchronization with an AV stream, metadata has been packetized usingvirtually the same size, and has been inserted as appropriate between AVstreams at virtually equal intervals.

The multiplexing section 1601 then sends this multiplex stream 1503 tothe information usage node 1506.

Next, the information usage section 1507 will be described using FIG.18. FIG. 18 is a block diagram of a conventional information usagesection.

The information usage section 1507 is provided with an extractionsection 1803 that performs separation and extraction, and output, of anAV stream 1801 and metadata 1802. The extraction section 1803 outputsthe separated and extracted AV stream 1801 and metadata 1802 to anaccess section 1804.

The access section 1801 stores the AV stream 1801 and metadata 1802input from the extraction section 1803 in a storage section 1508. Also,the access section 1804 outputs the AV stream 1805 and metadata 1806read from the storage section 1508 to a display section 1807. Thedisplay section 1807 displays either or both of the AV stream 1805 andmetadata 1806 input from the access section 1804.

Next, the processing of the information usage section 1507 will bedescribed using FIG. 19. FIG. 19 is a processing flowchart of aconventional information usage section.

The extraction section 1803 performs metadata parsing—that is, syntaxanalysis (ST1901). Then, execution of the processing of the accesssection 1804 and display section 1807 is performed (ST1902).

In this way, a conventional information processing system can display adescription relating to AV information, in addition to AV information,by means of the information usage node 1506 by having the informationprovision node 1501 transmit a multiplex stream multiplexing an AVstream and metadata to the information usage node 1506.

In recent years, a demand has arisen for various kinds of information tobe included in metadata, and for metadata to be processed coupled withan AV stream, rather than having metadata simply as ancillary data foran AV stream.

However, in the above-described conventional information processingsystem, metadata parsing cannot be carried out until all the metadatahas been acquired. For example, if metadata begins with <metadata>,metadata parsing cannot be carried out until data </metadata> indicatingthe end of the metadata arrives.

For this reason, the metadata processing time is closely tied to the AVstream display or processing time, and since an AV stream is processedin accordance with the metadata itself, processing cannot be starteduntil all the metadata has been received. Therefore, in a conventionalinformation processing system, there is a problem in that it isdifficult to process an AV stream in small units.

Also, metadata is distributed virtually uniformly in a multiplex stream.As a result, especially when the data quantity of metadata is large, alarge AV stream quantity must be read by the time all the metadata isread. Consequently, there are problems relating to inter-node responsetime delays and increased network traffic.

DISCLOSURE OF INVENTION

It is a first objective of the present invention to carry out data andprogram distribution for processing a segment comprising part of an AVstream, speeding up of response times, reduction of the necessarystorage capacity, and reduction of network traffic, by making possiblepartial execution of metadata.

Also, it is a second objective of the present invention to makeprocessing of a segment comprising part of an AV stream variable, andperform close synchronization between metadata and AV stream processingtimes, by implementing time synchronization of metadata and an AVstream.

Further, it is a third objective of the present invention to extend thedegree of freedom for designing metadata for processing an AV stream.

In order to meet the first objective, the present invention is providedwith a synchronization section which synchronizes a data stream segmentwith a unit of metadata corresponding to it, and a capsulization sectionwhich capsulizes a data stream packet and metadata unit packet aftersynchronization and generates a capsulized stream.

By this means, partial execution of metadata is made possible byreconfiguring metadata unit by unit and capsulizing it with the datastream. As a result, it is possible to carry out data and programdistribution for processing a segment comprising part of a data stream,speeding up of response times, reduction of the necessary storagecapacity, and reduction of network traffic.

In order to meet the second objective, the present invention is providedwith an extraction section which extracts from a capsulized stream acontent data stream and metadata for describing or processing thatcontent, a synchronization section which synchronizes metadata unitizedwith respect to an extracted data stream segment unit by unit with acontent data stream and the corresponding metadata unit, and aprocessing section which processes synchronized metadata unit by unit.

By this means, it is possible to make processing for a segmentcomprising part of a data stream variable, and perform closesynchronization between meta data and AV stream processing times.

In order to meet the third objective, the present invention uses astructured description for metadata and metadata units, and structureddescription re-format is performed from metadata to units and from unitsto metadata.

By this means, it is possible to extend the degree of freedom fordesigning metadata for processing a data stream. In addition, it ispossible for a structured description written in XML, etc., to be useddirectly as metadata.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an information processing system accordingto Embodiment 1 of the present invention;

FIG. 2 is a block diagram of an information processing section accordingto Embodiment 1;

FIG. 3A is a drawing showing an AV stream according to Embodiment 1;

FIG. 3B is a drawing showing metadata according to Embodiment 1;

FIG. 4A is a drawing showing DTD of XML of metadata according toEmbodiment 1;

FIG. 4B is a drawing showing DTD of XML of an MPU according toEmbodiment 1;

FIG. 5A is a drawing showing an instance of XML of metadata according toEmbodiment 1;

FIG. 5B is a drawing showing an instance of XML of an MPU according toEmbodiment 1;

FIG. 6 is a drawing showing the syntax of metadata according toEmbodiment 1;

FIG. 7 is a drawing for explaining the operation of a capsulizationsection according to Embodiment 1;

FIG. 8 is a block diagram of an information usage section according toEmbodiment 2 of the present invention;

FIG. 9 is a processing flowchart showing the metadata processingoperations of an information usage node according to Embodiment 2 of thepresent invention;

FIG. 10 is a block diagram of an information usage section according toEmbodiment 3 of the present invention;

FIG. 11 is a block diagram of an information usage section according toEmbodiment 4 of the present invention;

FIG. 12 is a block diagram of an information processing system accordingto Embodiment 5 of the present invention

FIG. 13 is a block diagram of an information processing sectionaccording to Embodiment 5;

FIG. 14 is a block diagram of an information usage section according toEmbodiment 4 of the present invention according to Embodiment 6;

FIG. 15 is a block diagram of a conventional information processingsystem;

FIG. 16 is a detailed drawing of a conventional information provisionsection;

FIG. 17 is a drawing showing the configuration of a conventionalmultiplex stream;

FIG. 18 is a detailed drawing of a conventional information usagesection; and

FIG. 19 is a processing flowchart for a conventional extraction section.

BEST MODE FOR CARRYING OUT THE INVENTION

With reference now to the attached drawings, embodiments of the presentinvention will be explained in detail below.

Embodiment 1

An information processing system according to Embodiment 1 of thepresent invention will be described below. FIG. 1 is a block diagram ofan information processing system according to Embodiment 1.

An information provision node 101 is provided with a storage section 102in which an AV stream and AV stream related metadata are stored. Themetadata is data that describes the related AV stream, or data forprocessing the metadata itself, or the like. Also provided in theinformation provision node 101 is an information provision section 101that multiplexes the AV stream and metadata stored in the storagesection 102 and generates and outputs a capsulized stream 103. Theinformation provision section 104 transmits the capsulized stream 103via a network 105 to an information usage node 106, which is anapparatus on the information receiving side.

Meanwhile, the information usage node 106 is provided with aninformation usage section 107 that extracts an AV stream and metadatafrom the capsulized stream 103 and executes predetermined processing onthem in order to use them. The information usage node 106 is alsoprovided with a storage section 108 that stores the AV stream andmetadata extracted by the information usage section 107. The informationusage section 107 reads the AV stream and metadata stored in the storagesection 108 in order to use them.

Next, the information provision section 104 will be described using FIG.2. FIG. 2 is a block diagram of an information provision sectionaccording to Embodiment 1.

The information provision section 104 is provided with an access section201 that reads an AV stream and metadata from the storage section 102.The access section 201 outputs an AV stream 202 and metadata 203 to asynchronization section 204.

The synchronization section 204 implements time synchronization for theAV stream 202 and metadata 203 read by the access section 201, andoutputs the synchronized AV stream 205 and metadata 206 to acapsulization section 207.

The capsulization section 207 capsulizes the synchronized AV stream 205and metadata 206, and transmits them to the information usage node 106as a capsulized stream 103.

Also, the present invention unitizes metadata to enable metadata to beexecuted in parts. Then, AV stream segments and corresponding metadataunits are synchronized, synchronized data stream packets and metadataunit packets are capsulized, and a capsulized stream is generated.

The operation of the information provision section 104 of the presentinvention will be described in detail below.

First, the AV stream 202 and metadata 203 stored in the storage section102 will be described using FIG. 3A and FIG. 3B.

The AV stream 202 has video PES packets 301 and audio PES packets 302interleaved to form a stream. In the present embodiment, a mode isdescribed whereby an AV stream 202 is scored in the storage section 102,but a mode is also possible whereby a video stream and audio stream arestored.

The metadata 203 is configured so as to have a plurality of MPUs(Metadata Processing Units) 303.

The thus configured metadata 203 and AV stream 202 are read from thestorage section 102 by the access section 201. Then the access section201 outputs the read AV stream 202 and metadata 203 to thesynchronization section 204.

On receiving the AV stream 202 and metadata 203, the synchronizationsection 204 first proceeds to processing for unitizing the metadata 203.Here, definitions of the metadata 203 and MPU 303 will be describedusing FIG. 4A and FIG. 4B. FIG. 4A and FIG. 4B are drawings showing DTDof XML. In FIG. 4A, 401 is a drawing showing a metadata definition(metadata.dtd) that defines the metadata 203. In FIG. 4B, the drawingindicated by reference numeral 402 shows an MPU definition (mpu.dtd)that defines an MPU 303.

The metadata definition 401 defines the metadata 203 as having one ormore MPUs 303. For the contents of an MPU 303, referencing the MPUdefinition 402 is defined.

The MPU definition 402 defines an MPU 303 as having one or moreelement_data items. For the contents of element_data, referencinguser_defined.dtd is defined. Also, the MPU definition 402 defines an MPU303 as having a serial number no assigned.

In this way, it is possible to include in an MPU 303 differentprocessing contents for each of various services according touser_defined.dtd. Thus, it is possible to extend the degree of freedomfor designing metadata for processing an AV stream.

Also, it is possible to include in an MPU 303 processing Contents not inaccordance with a transmission specification, according touser_defined.dtd. By this means, metadata can also be used for adifferent transmission specification, making it possible to providemetadata services that support a variety of transmission specifications.

Next, the unitization of metadata 203 will be described using FIG. 5Aand FIG. 5B. In FIG. 5A, the drawing indicated by reference numeral 501shows metadata (XML instance) whereby metadata 203 is given a structureddescription according to metadata definition 401, and the drawingindicated by reference numeral 502 shows an MPU (XML instance) wherebyan MPU 303 is given a structured description according to MPU definition402.

As described above, according to metadata definition 401, metadata 203is represented by a collection of MPU definitions 402. According to thismetadata definition 401. what gives a structured description of metadata203 is metadata (XML instance) 501. As can be seen from the drawing, themetadata (XML instance) 501 instance includes a plurality of MPUs 303.Also, metadata 203 is stored in the storage section 102 as metadata (XMLinstance) 501.

According to MPU definition 402, an MPU 303 is represented by acollection of metadata defined by user_defined.dtd. According to thisMPU definition 402, what gives a structured description of MPU 303 foreach MPU is MPU (XML instance) 502. As can be seen from the drawing, MPU(XML instance) 502 includes a plurality of user_defined.dtd items. Also,MPU 303 is stored in the storage section 102 as MPU (XML instance) 502.

An MPU 303 has contents <mpu> to </mpu>. That is to say, if there isinformation from <mpu> to </mpu>, the synchronization section 201 cangrasp MPU 303 contents and can perform MPU 303 processing. For thisreason, when picking out an MPU 303 front metadata 203, thesynchronization section 204 extracts the contents on the inside of a tagcalled an MPU tag (here, <mpu>) defined by an MPU definition 402.

By having metadata 203 composed of lower-level information MPUs 303 inthis way, the synchronization section 204 can perform metadata 203processing for each MPU 303, and also closely synchronize the AV data202 and metadata 203.

Next, the synchronization section 204 capsulizes metadata 203 sent fromthe access section 201 using the syntax shown in FIG. 6. FIG. 6 showsthe syntax of metadata according to Embodiment 1 and Embodiment 2.

In FIG. 6, metadata_type 601 is the metadata type such as positioninformation, content information, or program, metadata_subtype 602 isthe concrete metadata type such as GPS or structured description(MPEG-7). MPU_length 603 is the data length as a number of bytes fromimmediately after the MPU_length field to the end of the MPU. An MPU iscomposed of one or more PES packets, and is the regeneration unit ofmetadata divided when a Metadata Elementary Stream is encoded.media_sync_flag 604 is a flag indicating the presence or absence ofsynchronization between the AV stream and metadata, overwrite_flag 605is a flag indicating whether the previous metadata is to be overwritten.element_data_length 606 is the data byte length (M) of element_data 609.Start_time( ) 607 is the start time of a segment that is a part of theAV stream indicated by the metadata. duration( ) 608 is the continuationtime of a segment that is part of the AV stream indicated by themetadata. element_data 609 is the actual data of the metadata.

For the syntax shown in FIG. 6, coding uses syntax 610 from elsedownward even when the metadata data quantity is small and unitizationis not performed.

The synchronization section 204 capsulizes the AV stream segment forprocessing specified by the first packet's processing start time 607 andduration 608, and part of the metadata 203 corresponding to the segmentfor processing, as a capsulized stream (private PES).

When metadata 203 is PES-packetized, an MPU 303 is packetized togetherwith the AV stream segment first packet processing start time(start_time), duration( ) 608, and actual data of the metadata as anelement (element_data) in the metadata syntax shown in FIG. 6.

By this means, it is possible for an MPU 303 to have information formaintaining synchronization with the AV stream 202. Thus,synchronization is maintained between the MPU 303 and AV stream 202. Inthis way, metadata 203 operation can be determined on the informationprovision node 101 side.

Also, in Embodiment 1, an MPU 303 is composed of two packets—a first PESpacket 701 and a second FES packet 702—as shown in FIG. 7. Theoperations whereby the synchronization section 204 packetizes an MPU 303into private PBS packets and interleaves these with video PES packets301 and audio PES packets 302 in this case will be described using FIG.7. How many packets an MPU 303 is made into can be determinedarbitrarily according to the MPU 303 size and the packet size.

In the case of Embodiment 1, the first PES packet 701 and second PESpacket 702 are placed as private PES packets 708 earlier in time thanthe first packet 703 so that the first PES packet 701 and second PESpacket 702 are processed before the processing start time (start_time)705 of the first packet of the corresponding AV stream segment.

Also, the second PES packet 702 arrival time t 704 and the correspondingfirst packet 703 processing start time (start time) 705 difference at706 are assigned sufficient times for the information usage section 107,which is on the information receiving side, to generate an MPU 303 fromthe first PES packet 701 and second PES packet 702, and executeprocessing based on the contents of the generated MPU 303.

Then, the AV stream 205 and metadata 206 synchronized by thesynchronization section 201 in this way are input to the capsulizationsection 207.

The capsulization section 207 capsulizes the input AV stream 205 andmetadata 206, and transmits them as a capsulized stream 103.

As described above, according to Embodiment 1, metadata can bere-formatted unit by unit and capsulized with an AV stream by providinga synchronization section 204 that maintains synchronization of the AVstream and metadata, and a capsulization section 207 that capsulizesmetadata unit by unit with the AV stream. By this means, it becomespossible to perform partial execution of metadata, and to carry outprogram distribution for processing a segment comprising part of an AVstream, speeding up of response times, reduction of the necessarystorage capacity, and reduction of network traffic.

Moreover, according to Embodiment 1, by using a structured descriptionwritten using XML for metadata and metadata units, and performingstructured description re-format from metadata to units and from unitsto metadata, it is possible to provide extensibility for metadata forprocessing an AV stream, and extend the degree of freedom for designingmetadata. In addition, it is possible for a structured descriptionwritten in XML, etc., to be used directly as metadata.

Embodiment 2

Next, an information processing system according to Embodiment 2 of thepresent invention will be described. FIG. 8 is a block diagram of aninformation usage section 107 according to Embodiment 2.

The information usage section 107 is provided with an extraction section803 that performs separation and extraction, and output, of an AV stream801 and metadata 802. The extraction section 803 outputs the extractedAV stream 801 and metadata 802 to an access section 804.

The access section 804 records the AV stream 801 and metadata 802 in astorage section 108. Also, the access section 804 reads an AV stream 805and metadata 806 stored in the storage section 108, and outputs them toa synchronization section 807.

The synchronization section 807 performs time synchronization every MPU303 for the AV stream 805 and metadata 806 read by the access section804, and outputs them to a core processing section 808.

The core processing section 808 is provided with a display section 809.The display section 809 performs time synchronization and display of theinput synchronized AV stream 810 and metadata 811.

In this way, the information usage section 107 extracts an AV stream 801and metadata 802 from the capsulized stream 103 in the extractionsection 803. Then, in the synchronization section 807, the correspondingmetadata 802 unitized in accordance with AV stream 801 segments issynchronized with the AV stream 801 unit by unit. Then the synchronizedmetadata 811 and AV stream 810 a redisplayed unit by unit by the displaysection 809.

Next, the metadata processing operations or the information usage node106 will be described in detail using the flowchart in FIG. 9. First,the extraction section 803 extracts an AV stream and metadata from thereceived capsulized stream 103. In addition, the information usagesection 107 performs MPU 303 pursing (ST901). Next, in the informationusage section 107, a check is performed as to whether the MPUs 303 areto be merged and re-formatted as metadata 802 (ST902). Then, in theinformation usage section 107, a check is performed as to whether MPU303 execution is co be performed unit by unit (ST903).

If, in ST902 and ST903, the results confirmed by the information usagesection 107 are MPU merging and MPU execution, processing is executed bythe core processing section 808 (ST904). Then MPU merging is performedin the information usage section 107 (ST905). In Embodiment 2, thisprocessing is display processing, but it may also be conversionprocessing or transfer processing as in other embodiments to bedescribed hereafter.

Then, in the information usage section 107, judgment as to the advent ofan MPU time or number limit—that is, an event that indicates an MPUprocessing unit—is performed (ST906), and ST904 and ST905 are repeateduntil the advent of an event. Event information is given to softwarewhen providing universality, or is given to a terminal beforehand whenthe system is used in a fixed mode.

Then, in the information usage section 107, rendering—that is to say,formatting—of the metadata is performed from the MPUs collected togetherin ST906. Metadata formatted on the basis of this event is stored in thestorage section 108. Then the core processing section 808 reads thisformatted data and performs various kinds of processing.

In this way, it is possible not only to perform processing for each MPU,which is the minimum unit of processing, in ST904, but also to performprocessing based on data obtained by merging MPUs according to an event.

By this means, it is possible to set arbitrarily a unit for MPUprocessing according to an event, and therefore the length of AV datasegments for metadata processing can be made variable. That is to say,it is possible to process metadata for small AV data and to processmetadata for huge AV data. For example, it is possible to updatemetadata display in short cycles in a case such as a vehicle navigationsystem, and update metadata in long cycles in a case such as a newsprogram.

Also, by storing this metadata that has been formatted on the basis ofan event in the storage section 108, it is possible to read and processthis information by means of user operations.

If, in ST902 and ST903, the results confirmed by the information usagesection 107 are MPU merging and MPU non-execution, an MPU merge isperformed (ST908). Then, in the information usage section 107, judgmentas to the presence of an MPU time or number limit—that is, an eventrelated to completion of an MPU merge—is performed (ST909), and ST908 isrepeated until the occurrence of an event. Rendering of the metadata isthen performed from the MPUs collected together in processing P107.Then, in the information usage section 107, rendering—that is to say,formatting—of the metadata is performed from the MPUs collected togetherin ST906 (ST910). Metadata formatted on the basis of this event isscored in the storage section 108. Then the core processing section 808reads this formatted data and performs various kinds of processing.

In this way, it is possible not only to perform processing for each MPU,which is the minimum unit of processing, but also to perform processingbased on data obtained by merging MPUs according to an event.

If, in ST902 and ST903, the results confirmed by the information usagesection 107 are MPU non-merging and MPU execution, processing isexecuted sequentially (ST911). Then, in the information usage section107, judgment as to the presence of an MPU time or number limit—that is,an event that indicates an MPU processing unit—is performed (ST912), andST911 is repeated until the occurrence of an event.

In this way, it is possible to perform processing for each MPU, which isthe minimum unit of processing, and not to perform processing based ondata obtained by merging MPUs according to an event.

If, in ST902 end ST903, the results confirmed by the information usagesection 107 are MPU non-merging and MPU non-execution, no particularMPU-related processing is performed.

As described above, the extraction method can be changed as appropriateaccording to the contents contained in MPUs 303.

The operation of the information usage section 107 will now be describedbelow. The information usage section 107 extracts an AV stream 801 andmetadata 802 from the capsulized stream 103 input by the extractionsection 803, and outputs them to the access section 804. After recordingthe AV stream 801 and metadata 802 in the storage section 108, theaccess section 804 reads an AV stream 805 and metadata 806, and outputsthem to the synchronization section 807. The synchronization section 807performs time synchronization every MPU 303 for the AV stream 805 andmetadata 806 read by the access section 804, and outputs them to thecore processing section 808. In the core processing section 808, thedisplay section 809 performs time synchronization and display of theinput AV stream 810 and metadata 811.

As described above, according to Embodiment 2, close synchronization ofthe metadata and AV stream processing time can be performed by providingan extraction section 803 for separating and extracting an AV stream andmetadata, an access section 804 for reading and writing an AV stream andmetadata in a storage section 108, a synchronization section 807 forperforming synchronization of the read AV stream and metadataprocessing, and a display section 809, which is a core processingsection 808. By this means, it is possible to vary processing for asegment, which is part of an AV stream.

Also, information relating to the display method used by the displaysection 809 of the core processing section 808 can be provided asmetadata. Information relating to the display method includes positioninformation for displaying metadata related information, display sizeinformation, and display update information.

By this means, an appropriate method for displaying metadata can be sentto the information provision node 101 by the information usage node 106.As a result, metadata can be displayed appropriately by the informationusage node 106. Therefore, if metadata is an advertisement or the like,it is possible to make a specification that allows the advertisement tobe displayed at the desired time, and if metadata is information relatedto program descriptions, it is possible to display the descriptiveinformation so as not to interfere with images.

Moreover, according to Embodiment 2, by using a structured descriptionwritten using XML for metadata and metadata units, and performingstructured description re-format from metadata to units and from unitsto metadata, it is possible to extend the degree of freedom fordesigning metadata for processing an AV stream, and a structureddescription written in XML, etc., can be used directly as metadata.

Embodiment 3

Next, an in formation processing method according to Embodiment 3 of thepresent invention will be described. FIG. 10 is a block diagram of aninformation usage section 1001 according to Embodiment 3. Partsidentical to those that have already been described are assigned thesame reference numerals, and a description of these parts is omitted.

The information usage section 1001 according to Embodiment 3 has thecore processing section 808 of the information usage section 1001according to Embodiment 2 replaced by a core processing section 1002.Below, the information usage section 1001 will be described centering onthe core processing section 1002.

The core processing section 1002 is provided with a transfer section1003 and a capsulization section 1006.

The transfer section 1003 performs settings, such as a destinationsetting, for transferring an AV stream 810 and metadata 811 input fromthe synchronization section 807 to another information usage node. Thetransfer section 1003 performs time synchronization every MPU 303, andoutputs an AV stream 1004 and metadata 1005 to the capsulization section1006.

The capsulization section 1006 recapsulizes the input AV stream 1004 andmetadata 1005 and transmits them to another node as a capsulized stream1007. Since the capsulization section 1006 recapsulizes the AV stream1004 and metadata 1005 in this way, load sharing can be performed whilemaintaining close synchronization between the metadata and AV streamprocessing times.

The operation of the capsulization section 1006 is similar to that ofthe capsulization section 207 according to Embodiment 1, and so adetailed description will be omitted here.

The operation of the information usage section 1101 will now bedescribed below. The information usage section 1101 extracts an AVstream 801 and metadata 802 from the capsulized stream 103 input by theextraction section 803, and outputs them to the access section 804.After recording the AV stream 801 and metadata 802 in the storagesection 108, the access section 804 reads an AV stream 805 and metadata806, and outputs them to the synchronization section 807.

The synchronization section 807 performs time synchronization every MPU303 for the AV stream 805 and metadata 806 read by the access section304, and outputs them to the core processing section 1002. The coreprocessing section 1002 performs settings for transferring the AV stream810 and metadata 811 input by the transfer section 1003 to anotherinformation usage node, and performs time synchronization and output tothe capsulization section 1006 every MPU 303. The capsulization section1006 recapsulizes the input AV stream 1004 and metadata 1005 andtransmits them to another node as a capsulized stream 1007.

By configuring the information usage section 1001 as described above, itis possible for the transfer section 1003 to perform settings fortransferring the AV stream 810 and metadata 811 input from thesynchronization section 807 to another information usage node, performtime synchronization and output to the capsulization step 23 every MPU303, and for the capsulization section 1006 to recapsulize the AV stream1004 and metadata 1005 input from the transfer section 1003 and transmitthem to another node as a capsulized stream 1007.

As described above, according to Embodiment 3, it is possible for loadsharing to be performed while maintaining close synchronization betweenthe metadata and AV stream processing times, and also to make processingfor a segment comprising part of a data stream variable, by providing inthe information usage section 1001 an extraction section 803 forseparating and extracting an AV stream and metadata, an access section804 for reading and writing an AV stream and metadata in a storagesection 108, a synchronization section 807 for performingsynchronization of the read AV stream and metadata processing, and, inthe core processing section 1002, a transfer section 1003 and acapsulization section 1006.

Moreover, according to Embodiment 3, it is also possible for informationabout the processing methods of the transfer section 1003 andcapsulization section 1006, or a processing program itself, to be mademetadata. Processing method here refers to processing for changing theplace where metadata is inserted according to the transfer destination,for instance. By this means, it is possible for the informationprovision node 101 to send appropriate information for transferring andcapsulizing metadata to the information usage node 106. As a result, itis possible for metadata to be transferred and capsulized appropriatelyby the information usage node 106.

Embodiment 4

Next, an information processing system according to Embodiment 4 of thepresent invention will be described. FIG. 11 is a block diagram of aninformation usage section 1101 according to Embodiment 4. Partsidentical to those that have already been described are assigned thesame reference numerals, and a description of these parts is omitted.

The information usage section 1101 according to Embodiment 4 isequivalent to the information usage section 107 according to Embodiment2 or the information usage section 1001 according to Embodiment 3provided with a conversion section 1102. Below, the information usagesection 1101 will be described centering on the conversion section 1102.

The conversion section 1102 converts an AV stream 810 in accordance withmetadata 811, and outputs the result to the core processing section 1105as a T-AV stream 1103 and T-metadata 1104. The conversion referred tohere is color conversion according to the transmission destinationterminal or display position, graphic information format conversionaccording to the transmission destination terminal or display position,or conversion of the voice format to an MP3 or portable phone formataccording to the transmission destination terminal.

The core processing section 1105 operates in the same way as either thecore processing section 808 shown in Embodiment 2 or the core processingsection 1002 shown in Embodiment 3.

If the core processing section 1105 is core processing section 808, thecore processing section 1105 is provided with a display section 809. Inthis case the display section 809 performs display while carrying outtime synchronization of the input T-AV stream 1103 and T-metadata 1104.

If the core processing section 1105 is core processing section 1002, thecore processing section 1105 is provided with a transfer section 1003and capsulization section 1006. In this case, the transfer section 1003performs settings for transferring the T-AV stream 1103 and T-metadata1104 input by the transfer section 1003 to another information usagenode, and performs time synchronization and output to the capsulizationsection 1006 every MPU 303. The operation of the capsulization sectionaccording to Embodiment 3 is similar to that of the capsulizationsection 207 of Embodiment 1.

The operation of the information usage section 1101 will now bedescribed below. The information usage section 1101 extracts an AVstream 801 and metadata 802 from the capsulized stream 103 input by theextraction section 803, and outputs them to the access section 804.After recording the AV stream 801 and metadata 802 in the storagesection 108, the access section 804 reads an AV stream 805 and metadata806, and outputs them to the synchronization section 807. Thesynchronization section 807 performs time synchronization every MPU 303for the AV stream 805 and metadata 806 read by the access section 804,and outputs them to the conversion section 1102. The conversion section1102 then converts AV stream 810 according to metadata 811, and outputsthe results to the core processing section 1105 as a T-AV stream 1103and T-metadata 1104.

Then, if the core processing section 1105 is the core processing section808 according to Embodiment 2, the display section 809 performs displaywhile carrying out time synchronization of the input T-AV stream 1103and T-metadata 1104. If the core processing section 1105 is the coreprocessing section 1002 according to Embodiment 1, the transfer section1003 performs settings for transferring the T-AV stream 1103 andT-metadata 1304 input by the transfer section 1003 to anotherinformation usage node, and performs time synchronization and output tothe capsulization section 1006 every MPU 303. The capsulization section1006 recapsulizes the input T-AV stream 1103 and T-metadata 1104, andtransmits them as a capsulized stream 1007.

As described above, according to Embodiment 4, it is possible for theplace where conversion processing is performed according to metadata tobe made variable by having the information usage section 1101 providedwith an extraction section 803 for separating and extracting an AVstream and metadata, an access section 804 for reading and writing an AVstream and metadata in a storage section 108, a synchronization section807 for performing synchronization of the read AV stream and metadataprocessing, and, as the core processing section 1105, a usage programcomposed of a display section 809 or a transfer section 1003 andcapsulization section 1006. The place where conversion processing isperformed may be, for example, a server, terminal, network node(gateway), or the like.

Moreover, according to Embodiment 4, it is possible to make processingfor a segment comprising part of an AV stream variable. Also, AV streamand metadata conversion can be made possible.

Furthermore, according to Embodiment 4, performing further processing ona converted AV stream and metadata can be made possible.

Still further, according to Embodiment 4, by using a structureddescription written using XML for metadata and metadata units, andperforming structured description re-format from metadata to units andfrom units to metadata, it is possible to extend the degree of freedomfor designing metadata for processing an AV stream, and a structureddescription written in XML, etc., can be used directly as metadata.

In addition, according to Embodiment 4, it is possible for informationrelating to methods for processing metadata in the core processingsection 1105—the display method, transfer method, and capsulizationmethod—to be made metadata.

Embodiment 5

Next, an information processing system according to Embodiment 5 of thepresent invention will be described. FIG. 12 is a block diagram of aninformation processing system according to Embodiment 5. Parts that havealready been described are assigned the same reference numerals.

Embodiment 5 has a configuration that omits the processing forsynchronizing an AV stream and metadata from the information provisionsection 104 according to Embodiment 1. By omitting synchronizationprocessing in this way, when synchronization of an AV stream andmetadata is not necessary, processing speed can be increased by omittingsynchronization processing and the configuration can be simplified.Examples of cases where synchronization of an AV stream and metadataneed not be performed include cases where metadata is sent all togetheras with header information and processing need only be performed unit byunit, where it is sufficient for metadata to be synchronized implicitlywith the AV stream, where it is sufficient for predetermined control tobe performed by the terminal on the information usage side, and wheremetadata need not be processed in real time.

The configuration of an information processing system according toEmbodiment 5 will now be described below.

An information provision node 1201 is provided with a storage section102 in which an AV stream and AV stream related metadata are stored. Themetadata is data that describes the related AV stream, or data forprocessing the metadata itself, or the like. Also provided in theinformation provision node 1201 is an information provision section 1204that capsulizes the AV stream and metadata stored in the storage section102 and generates and outputs a capsulized stream 1203. The informationprovision section 1204 transmits the capsulized stream 1203 via anetwork 105 to an information usage node 1206, which is an apparatus onthe information receiving side.

Meanwhile, the information usage node 1206 is provided with aninformation usage section 1207 that extracts an AV stream and metadatafrom the capsulized stream 1203 and executes predetermined processing onthem in order to use them. The information usage node 1206 is alsoprovided with a storage section 108 that stores the AV stream andmetadata extracted by the information usage section 1207. Theinformation usage section 1207 reads the AV stream and metadata storedin the storage section 108 in order to use them.

Next, the information provision section 1204 will be described usingFIG. 13. FIG. 13 is a block diagram of an information provision sectionaccording to Embodiment 5.

The information provision section 1204 is provided with an accesssection 1301 that reads an AV stream and metadata from the storagesection 102. The access section 1301 outputs an AV stream 1302 andmetadata 1303 to a unitization section 1304.

The unitization section 1304 reforms metadata 1306 read by the accesssection 1301 into MPUs 303, and also outputs the synchronized AV stream1305 and metadata 1306 read by the access section 1301 to acapsulization section 1307.

The capsulization section 1307 capsulizes the input AV stream 1305 andmetadata 1306, and transmits them to the information usage node 1206 asa capsulized stream 1203.

In Embodiment 5, as in Embodiment 1, metadata is unitized to enable itto be executed in parts. Then, the AV stream and metadata units arepacketized, data stream packets and metadata unit packets arecapsulized, and a capsulized stream is generated.

The operation of the information provision section 1204 of the presentinvention will be described in detail below. Details of the AV stream1302 and metadata 1303 stored in the storage section 102 are the same asfor the AV stream 202 and metadata 203 according to Embodiment 1, so adescription of these will be omitted here.

With the above-described configuration, metadata 1303 and an AV stream1302 are read from the storage section 102 by the access section 1301.Then the access section 1301 outputs the read AV stream 1302 andmetadata 1303 to the unitization section 1304.

On receiving the AV stream 1302 and metadata 1303, the unitizationsection 1304 first proceeds to processing for unitizing the metadata1303.

Definitions of the metadata 1303 and MPUs 303 are the same as for themetadata 203 according to Embodiment 1 and the MPUs 303 described inEmbodiment 1, so a descript ion of these will be omitted here. Also, theprocess of unitization of the metadata 1303 is the same as forunitization of the metadata 203 according to Embodiment 1, so adescription of this will be omitted here.

According to metadata definition 101 shown in FIG. 4A, metadata 1303 isrepresented by a collection of MPU definitions 402. Therefore, metadata1303 is given a structured description by means of metadata definition401, and is stored in the storage section 102 as metadata (XML instance)501 shown in FIG. 5A.

Also, according to MPU definition 402 shown in FIG. 4B, An MPU 303 isrepresented by a collection of metadata defined by user_defined.dtd.Therefore, MPUs 303 are given a structured description for each MPU bymeans of MPU definitions 402, and are stored in the storage section 102as MPU (XML instance) 502 shown in FIG. 5B.

An MPU 303 has contents <mpu> to </mpu>. That is to say, if there isinformation from <mpu> to </mpu>, the unitization section 1304 can graspMPU 303 contents and can perform MPU 303 processing. For this reason,when picking out an MPU 303 from metadata 1303, the unitization section1304 extracts the contents on the inside of a tag called an MPU tag(here, <mpu>) defined by an MPU definition 402.

By having metadata 1303 composed of lower-level information MPUs 303 inthis way, the unitization section 1304 can perform metadata 1303processing for each MPU 303. By this means, the unitization section 1304can process AV data 1302 and metadata 1303 unit by unit.

Next, as in Embodiment 1, the capsulization section 1307 capsulizesmetadata 1306 sent from the unitization section 1304 using the syntaxshown in FIG. 6.

The capsulization section 1307 then capsulizes the AV stream segment forprocessing specified by the first packet's processing start time 607 andduration 608, and part of the metadata 1303 corresponding to the segmentfor processing, as a capsulized stream (private PES).

The unitization section 1304 then packetizes MPUs 303 into private PESpackets and interleaves these with video PES packets and audio PESpackets.

Then the capsulization section 207 capsulizes the input AV stream 1305and metadata 1306, and transmits them as a capsulized stream 1203.

As described above, according to Embodiment 5, metadata can bere-formatted unit by unit and capsulized with an AV stream by providinga unitization section 1304 that unitizes the AV stream and metadata, anda capsulization section 1307 that capsulizes the metadata unit by unitwith the AV stream. By this means, it becomes possible to performpartial execution of metadata, and to carry out program distribution forprocessing a segment comprising part of an AV stream, speeding up ofresponse times, reduction of the necessary storage capacity, andreduction of network traffic.

Moreover, since Embodiment 5, unlike Embodiment 1, omits synchronizationprocessing, when synchronization of an AV stream and metadata is notnecessary, processing speed can be increased by omitting synchronizationprocessing and the configuration can be simplified.

Embodiment 6

Next, an information processing system according to Embodiment 6 of thepresent invention will be described. FIG. 14 is a block diagram of aninformation usage section 1207 according to Embodiment 6.

Embodiment 6 has a configuration that omits the processing forsynchronizing an AV stream and metadata from the information usagesection 107 according to Embodiment 2. By omitting synchronizationprocessing in this way, when synchronization of an AV stream andmetadata is not necessary, processing speed can be increased by omittingsynchronization processing and the configuration can be simplified.Examples of cases where synchronization of an AV stream and metadataneed not be performed include cases where metadata is sent all togetheras with header information and processing need only be performed unit byunit, where it is sufficient for metadata to be synchronized implicitlywith the AV stream, where it is sufficient for predetermined control tobe performed by the terminal on the information usage side, and wheremetadata need not be processed in real time.

The configuration of an information processing system according toEmbodiment 6 will now be described below.

An information usage section 1207 is provided with an extraction section1403 that extracts and outputs an AV stream 1401 and metadata 1402 froman input capsulized stream 1203. The extraction section 1403 outputs theextracted AV stream 1401 and metadata 1402 to an access section 1404.

The access section 1404 records the AV stream 1401 And metadata 1402 ina storage section 108. Also, the access section 1404 reads an AV stream1405 and metadata 1406 stored in the storage section 108, and outputsthem to a core processing section 1407.

The core processing section 1407 operates in the same way as the coreprocessing section 808 shown in Embodiment 2. If the core processingsection 1105 is core processing section 808, the core processing section1407 is provided with a display section 1408. In this case the displaysection 1408 displays the input AV stream 1405 and metadata 1406.

In this way, the information usage section 1207 extracts an AV stream1401 and metadata 1402 from the capsulized stream 1203 in the extractionsection 1403. Then, the display section 1408 displays metadata 1406 andAV stream 1405 unit by unit.

The operation of the information usage section 1207 will now bedescribed below. The information usage section 1207 extracts an AVstream 1401 and metadata 1402 from the capsulized stream 1203 input bythe extraction section 1403, and outputs them to the access section1404. After recording the AV stream 1401 and metadata 1402 in thestorage section 108, the access section 1404 reads an AV stream 1405 andmetadata 1406, and outputs them to the core processing section 1407. Inthe core processing section 1407, the display section 1408 displays theinput AV stream 1405 and metadata 1406.

As described above, according to Embodiment 6, it is possible to makeprocessing for a segment comprising part of a data stream variable byproviding an extraction section 1403 for separating and extracting an AVstream and metadata, an access section 1404 for reading and writing anAV stream and metadata in a storage section 108, and a display section1408, which is a core processing section 1407.

Moreover, since Embodiment 6, unlike Embodiment 2, omits synchronizationprocessing, when synchronization of an AV stream and metadata is notnecessary, processing speed can be increased by omitting synchronizationprocessing and the configuration can be simplified.

Embodiment 6 has been described as having a configuration in which thesynchronization section 807 is omitted from Embodiment 2, but aconfiguration may also be used in which the synchronization section 807is omitted from Embodiment 3 or 4.

In Embodiment 3 to Embodiment 6, each processing section is configuredby having all or part of the respective operations stored as a program(software) on a computer-readable storage medium such as a CD-ROM orDVD, and having the operations of each processing section performed bythe CPU of a computer, or the like, by having a computer read theprogram.

A mode is also possible whereby all or part of the operations of eachprocessing section are stored on a storage medium on communication meanssuch as the Internet or the like as a program (software), the program isdownloaded to an information terminal via the Internet or the like, andthe operations of each processing section are performed by theinformation terminal.

A mode is also possible whereby each processing section is configuredusing dedicated hardware.

In Embodiment 1 to Embodiment 6, descriptions have used an AV stream asa content data stream with timewise continuity, but the sane kind ofeffects as in the above-described embodiments can be obtained with notan AV stream but another stream, file, or small-volume information, aslong as its use as a stream is considered useful.

In Embodiment 1 to Embodiment 6, metadata definitions and MPUdefinitions are performed using DTD of XML, but XML RDF or XML Schemamay be used, or other definition means may also be used.

In Embodiment 1 to Embodiment 6, packetization has been described withMPEG-2 system PES packets, but an MPEG-1 system, MPEG-4, SMPTE AncillaryData Packet, or another transmission, format, streaming format, or fileformat may also be used.

In Embodiment 1 to Embodiment 6, private PES has been used for thedescription of the transmission layer for sending metadata, but metadataPES, MPEG-7 PES, MPEG-2 PSI (Program Specific Information) Section(so-called carousel) promised for the future may also be used as atransmission layer.

In Embodiment 1 to Embodiment 4, as a synchronization variation, one MPUmay also be inserted repeatedly to enable the necessary data to bereceived when starting reception midway.

In Embodiment 1 to Embodiment 6, the network 105 or 1505 may be aterrestrial broadcasting network, a satellite broadcasting network, acable television network, a line switching network, a packet switchingnetwork, an ATM, the Internet, or another network, package medium, harddisk, memory, or the like.

This application is based on the Japanese Patent Application No. HEI11-200095 filed on Jul. 14, 1999, entire content of which is expresslyincorporated by reference herein.

INDUSTRIAL APPLICABILITY

As described above, according to the present invention, firstly, partialexecution of metadata is made possible, and it is possible to carry outprogram distribution for processing a segment comprising part of an AVstream, speeding up of response times, reduction of the necessarystorage capacity, and reduction of network traffic, by reconfiguringmetadata unit by unit and capsulizing it with an AV stream; secondly,close synchronization between metadata and AV stream processing timescan be performed by making processing of a segment comprising part of anAV stream variable; and thirdly, it is possible to extend the degree offreedom for designing metadata for processing an AV stream, and to use astructured description written in XML, etc., directly as metadata, byusing a structured description by means of XML for metadata and metadataunits, and performing structured description re-format from metadata tounits and from units to metadata.

What is claimed is:
 1. A decoding method for decoding an audiovisualstream that is divided into a plurality of segments, the decoding methodcomprising: obtaining metadata that specify a segment out of theplurality of segments, the metadata being described in a structureddescription; obtaining the segment specified by the metadata; deriving astart time for rendering the segment from the metadata; and decoding thesegment based on the metadata to generate decoded segment data beforethe start time.
 2. The decoding method according to claim 1, wherein, inthe decoding, the segment is decoded in response to an event occuring.3. The decoding method according to claim 1, wherein the structureddescription is defined by extensible markup language schema.
 4. Thedecoding method according to claim 1, wherein the metadata are obtaineda predetermined time before the segment is obtained.
 5. The decodingmethod according to claim 1, further comprising: obtaining firstmetadata and a first segment specified by the first metadata; obtainingsecond metadata and a second segment specified by the second metadataafter both of the first metadata and the first segment are obtained; anddecoding the second segment to be directly subsequent to the firstsegment.
 6. The decoding method according to claim 1, wherein theplurality of segments each comprise a part of the audiovisual stream,and processing times of the plurality of segments are variable.
 7. Adecoding apparatus for decoding an audiovisual stream that is dividedinto a plurality of segments, the decoding apparatus comprising: aprocessor; and a memory storing instructions that, when executed by theprocessor, cause the processor to perform operations including:obtaining metadata that specify a segment out of the plurality ofsegments, the metadata being described in a structured description;obtaining the segment specified by the metadata; deriving a start timefor rendering the segment from the metadata; and decoding the segmentbased on the metadata to generate decoded segment data before the starttime.